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EXECUTIVE  SUMMARY 


A  model  for  characterizing  situation  awareness  (SA)  was  tested  by  applying  its  elements  to 
performance  data  gathered  on  Marine  Corps  Infantrymen  from  both  training  and  operational 
deployment.  The  objective  was  to  determine  whether  model-based  factors  of  SA  could  be  discerned 
strictly  from  operational  settings,  as  contrasted  with  controlled  laboratory  conditions.  The  method 
involved  the  collection  of  “critical  incident”  reports  of  serious  military  failures,  gathered  from 
training  and  operations,  which  could  be  attributed  to  decision-making  breakdowns.  Reports  were 
examined  for  instances  of  model-relevant  features  and  results  were  analyzed  as  frequency  plots.  Such 
an  empirical  approach  to  performance  analysis  has  the  benefits  of  strong  contextual  validity  (data 
gathered  amid  free-running  mission  activities)  and  clearly  documented  military  consequences.  Initial 
results  showed  interpretable  differences  in  the  type  of  SA  breakdowns  that  could  account  for  these 
incidents,  categorized  by  level  of  leadership,  mission,  setting  (i.e.,  training  or  operations),  and  other 
contributing  factors. 
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BACKGROUND 


Understanding  and  responding  appropriately  to  chaotic  events,  and  anticipating  their  consequences 
in  real  time,  are  essential  skills  to  military  success  and  survival.  These  skills  require  situation 
awareness  (SA):  the  capability  of  the  warfighter  to  apply  the  proper  action  at  the  proper  time  for  both 
tactical  and  strategic  advantage.  Situation  awareness  is  an  active  topic  of  analysis  for  military 
operations  support.  The  assumption  is  that  superior  SA  will  increase  the  probability  (although  not  the 
certainty)  of  victory  through  improved  assessment  of,  and  response  to,  combat  events. 

Although  SA  has  been  examined  from  a  variety  of  perspectives  (e.g.,  Dominguez,  1994;  Fracker, 
1998),  the  practical  foundation  of  SA  as  a  research  construct  is  largely  credited  to  the  original  work 
of  Mica  Endsley,  who  provided  both  a  formal  model  and  a  set  of  analytical  methods  (e.g.,  Endsley, 
1987,  1990,  1995)  relevant  to  evaluating  warfighting  performance. 

The  value  of  good  SA  to  warfighter  viability  is  hard  to  question.  As  stated  by  Endsley  and  others, 
situation  awareness  is  “knowing  what  is  going  on  around  you,”  a  description  familiar  to  anyone  in 
the  military.  The  Endsley  model  (e.g.,  Endsley,  1993)  proposes  three  levels  of  SA:  “the  perception  of 
the  elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near  future.”  These  characteristics  are  more 
formally  labeled  as  Level  1  (Perception),  Level  2  (Comprehension),  and  Level  3  (Projection), 
respectively  (Endsley,  2000).  These  model  elements  can  characterize  both  experience  (e.g.,  from 
simply  perceiving  basic  stimuli  by  the  new  soldier  to  the  rapid  and  complex  field  assessments  of  the 
combat  veteran)  and  native  ability.  . 

Good  SA  cannot  guarantee  good  decision  performance,  of  course.  Errors  can  occur  because  of 
procedural  constraints  (such  as  tactics  or  rules  of  engagement)  or  poor  execution  of  an  appropriate 
response.  Furthermore,  SA  is  influenced  by  the  time  that  an  individual  has  to  process  information  and 
the  fact  that  most  military  situations  are  dynamic,  which  means  that  SA  must  continually  adapt. 
Nevertheless,  although  poor  SA  does  not  preclude  good  outcomes  (e.g.,  the  luck  encountered  in  the 
“fog  of  war”),  it  is  reasonable  to  believe  that  good  SA  improves  the  likelihood  of  good  operational 
decision  making  (Adams,  Tenney,  and  Pew,  1995;  Smith  and  Hancock,  1994). 

SITUATION  AWARENESS  RESEARCH  APPROACH 

Most  early  SA  studies  involved  aviation  tasks  (Endsley,  1987,  1993;  Fracker,  1988;  Hartman  and 
Secrist,  1991;  Sarter  and  Woods,  1991;  Taylor,  1990),  although  examination  of  ground  warfighter 
performance,  and  development  of  measurement  methods  to  address  it,  has  also  been  conducted  by 
the  Army  (e.g.,  Endsley,  et  ah,  1999;  Matthews  and  Beal,  2002).  Such  studies  have  focused  on  SA 
for  medium  and  large-size  (e.g.,  platoon,  company,  brigade)  forces  and  on  officer  level  leadership 
(e.g.,  Strater,  et  ah,  2001;  Strater,  Jones,  and  Endsley,  2001;  Matthews,  et  al.Strater,  and  Endsley, 
2004;  Warner,  et  al.Finklaire,  and  Pacey,  2001).  While  this  work  has  generated  important  insights 
into  SA  measurement  and  understanding,  certain  limitations  of  the  research  model  and  its  focus  need 
to  be  highlighted  to  place  current  knowledge  into  context: 

•  Tasks  related  to  platoon-level  operations  (and  above)  typically  involve  the  aggressive  use  of 
information  technologies  for  intelligence  gathering,  communication,  etc.  and — because  these 
technologies  are  the  fundamental  tools  for  developing  combat  situation  awareness  (CSA) — the 
content  of  such  technologies  have  dominated  SA  investigations  (e.g.,  Bolstad  and  Endsley, 
2000;  Farrell,  Jameson,  and  Stoneking,  2003;  Jameson,  2001). 

•  Most  SA  research  involving  ground  forces  has  relied  on  simulations  or  other  structured 
exercises  for  experimental  control  of  conditions  and  manipulations  (e.g.,  Matthews,  et  ah, 

2000;  Strater,  et  ah,  2001). 
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•  The  primary  SA  data  collection  method,  and  most  of  its  variants,  requires  some  form  of 
intrusion  into  the  test  environment.  Typically,  SA  experiment  design  involves  interruption  of 
the  mission  under  study  (e.g.,  “freezing”  a  simulation)  and  administration  of  a  set  of  questions 
regarding  participants’  knowledge  of  tactical  conditions.  Interruptions  are  made  at  multiple 
random  points  during  an  experiment  session  in  order  to  generate  the  performance  data  set  (e.g., 
Endsley,  et  al..,  1999;  Matthews,  et  al..,  2000).  The  validity  and  scientific  impact  of  results 
obtained  in  this  way  may  be  limited,  as  such  “clean”  conditions  are  never  obtainable  in  the 
field,  and  the  temporal  dynamic  of  operational  missions  is  fundamentally  different  from 
simulations  marked  by  periodic  interruptions. 

MILITARY  APPLICATION  OF  SA  RESEARCH 

SA  research  has  proven  valuable  to  the  military  command  levels  to  which  it  has  been  applied.  SA 
is  important  both  strategically  and  tactically,  however,  and  studies  of  SA  at  the  individual — 
especially  junior  officer  and  enlisted — level  are  no  less  critical,  especially  since  the  brunt  of  current 
military  operations  involve  small  units  and  small  unit  leaders.  This  form  of  combat  relies  more  on 
raw  sensory  experience  as  input,  with  less  use  of  technology  and  information  systems.  It  is 
characterized  by  shorter  response  times,  more  immediate  feedback,  and  more  rapid  fluctuations  in 
relevant  conditions. 

The  work  reported  here  was  motivated  by  the  lack  of  significant  SA  analyses  in  the  literature 
regarding  Marine  Corps  units  at  a  time  when  Marine  operations  are  both  protracted  and  intense.  The 
decision-making  performance  of  the  patrolling  Marine  is  highly  dependent  on  SA  (e.g.,  USMC, 
2005),  so  SA  research  focused  on  this  population  and  its  missions  could  improve  combat 
effectiveness  and  save  lives.  Furthermore,  the  Marine  Corps  continues  to  evaluate  a  doctrine  of 
“distributed  operations”  (Goulding,  2005)  that  will  place  more  decision-making  responsibility  on 
lower  enlisted  ranks — individuals  with  the  least  technology  support  and  the  most  unpredictable 
operating  environments.  The  current  work  therefore  focused  on  small  unit  missions  and  junior 
leaders  for  reasons  of  relevance  to  both  current  and  projected  Marine  Corps  operations. 

MOTIVATION  FOR  THE  CURRENT  STUDY 

Discussions  with  U.S.  Army  researchers  and  testing  agencies  revealed  anecdotal  reports  about  the 
limited  utility  of  current  SA  measurement  methods  (i.e.,  structured  environments  and  scenario 
interruptions)  when  applied  to  deployed  operations  or  field  training.  Such  reports  tend  to  call  into 
question  the  applicability  of  current  SA  experiment  designs  to  practical  operations.  It  is  reasonable, 
however,  to  expect  that  a  model  of  human  performance — that  is,  one  useful  for  both  explaining 
observed  phenomena  and  predicting  future  phenomena — should  find  instantiation  in  real-world 
environments.  A  key  requirement  of  the  current  study,  therefore,  was  to  examine  only  continuous, 
“free  play”  mission  activities  that  had  observable,  negative  outcomes,  to  ensure  military  relevance.  In 
other  words,  this  study  focused  on  SA  problems  that  occur  during  actual  mission  performance,  rather 
than  on  problems  more  typically  generated  through  structured  scenario  control  in  research  settings. 

An  alternate  method  to  scenario  interruption  for  performance  measurement  can  be  found  through 
critical  incident  reporting  (e.g.,  Klein,  et  al.Calderwood,  and  Macgregor,  1989)  used  in  the  Aviation 
Safety  Reporting  System  (NASA;  2006).  The  Aviation  Safety  Reporting  System  (ASRS)  is  a  data 
collection  tool  and  database  that  utilizes  anonymous  pilot  self-reporting  of  flight  incidents  (e.g., 
hazards  that  did  not  result  in  overt  mishaps)  to  help  identify  problematic  characteristics  of  the 
aviation  environment.  Analysis  of  the  ASRS  database  is  used  to  guide  structured  research  of  specific 
issues,  to  change  aviation  policy,  and  to  refine  pilot  training.  This  study  utilizes  a  similar  incident 
reporting  technique  in  support  of  Marine  decision-making  research.  A  useful  feature  of  this 
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performance  measurement  approach  is  that  it  can  be  applied  to  both  training  and  operational 
activities  without  modification.  An  incident  reporting  technique  is  used  to  extract  SA  data  from 
operational  events,  and  to  evaluate  the  potential  of  such  an  approach  as  an  SA  investigation  tool  to 
supplement  the  results  of  more  structured  research.  The  three  goals  of  the  study  reported  here  were 
to: 

•  Characterize  and  quantify  the  SA  performance  of  small  unit  leaders  for  current  combat 
missions  (i.e.,  CSA)  conducted  by  the  Marine  Corps 

•  Utilize  an  incident  report  approach,  developed  for  this  purpose,  to  CSA  performance 
measurement 

•  Compare  SA  error  profiles  from  both  training  and  operational  deployment  settings 

MARINE  CORPS  MISSION  TRAINING 

The  Marine  Corps  conducts  mission-specific  weapons  and  tactics  training  for  all  units  prior  to 
operational  deployment.  Data  collection  for  this  study  was  initiated  in  2005  during  Stability  and 
Security  Operations  (SASO)  training  conducted  at  March  Air  Reserve  Base,  and  continued  as  this 
training  was  transitioned  to  the  Marine  Air  Ground  Combat  Center  (MAGCC),  Twentynine  Palms, 
CA,  in  2007,  as  part  of  the  Mojave  Viper  training  syllabus. 

SASO  included  missions  and  tactics  required  of  small  units  operating  among  civilian  populations 
(specifically,  Operation  Iraqi  Freedom).  The  Marine  Corps  scheduled  SASO  training  for  battalions 
just  prior  to  their  operational  deployment.  The  syllabus  focused  on  small  units,  operating  on  the 
move  in  ill-defined  or  rapidly  changing  circumstances,  and  typically  far  from  other  assistance.  The 
environment  was  designed  to  be  representative  of  operating  conditions  in  Iraq. 

A  major  benefit  of  data  collected  from  SASO  and,  later,  Mojave  Viper,  was  that  training  activities 
were  divided  into  mission-specific  events  that  aided  focused  performance  measurement.  The  high 
instructor- to-student  ratio  also  offered  a  wealth  of  operationally  experienced  observers  and  subject 
matter  experts  (SMEs)  to  provide  performance  assessment.  SASO  training  missions  consisted  of: 

•  Patrol  (on  foot) 

•  Mounted  Patrol  (which  included  mechanized  patrol) 

•  Cordon  and  Knock  -  an  organized  effort  to  surround  and  secure  a  building,  and  then  to  control, 
search,  or  question  the  building  occupants 

•  Urban  Assault  -  securing  and  occupying  a  building  or  area  by  force,  when  occupied  by  hostile 
forces.  This  is  the  only  mission  with  an  a  priori  offensive  orientation,  i.e.,  the  intent  is  one  of 
engaging  and  prevailing  over  a  hostile  threat 

•  Checkpoint  Operations  -  intended  to  set  up  vehicle  “stop  and  inspect”  security  stations 

•  Convoy  -  multi-vehicle  movement  of  personnel  and  materiel,  often  through  uncontrolled  or 
hostile  areas 

•  Firm  Base  -  events  occurring  for  the  defense  and  operation  of  a  fixed  secure  area,  from  which 
patrols  and  other  missions  are  then  deployed 

•  FEX  -  a  final  exercise  involving  a  full  range  of  missions  and  problems  lasting  over  72 
continuous  hours 

To  establish  a  common  basis  for  evaluation,  all  component  training  missions  used  for  this  study 
(including  operational  reports  provided  from  Iraq)  were  mapped  to  the  original  SASO  syllabus 
structure. 
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METHOD 


The  study  relied  primarily  on  instructor  evaluations  of  SA  performance,  gathered  as  part  of  their 
normal  training  observation  duties.  Instructors  were  used  as  the  data  collection  source  because  they 
best  understood  what  constituted  militarily  meaningful  decision-making  “errors.”  Equally  important, 
they  understood  what  mistakes  were  not  important.  The  instructor  cadre  therefore  provided  an 
effective,  operationally  relevant  filter  for  performance  measurement,  superior  in  relevance  and 
validity  to  the  judgment  of  untrained  researchers.  The  data  set  was  supplemented  by  reports  solicited 
from  Marines  on  deployment  in  Iraq,  under  sponsorship  of  the  Marine  Corps  Training  and  Education 
Command  (TECOM). 

DATA  SOURCES 

A  CSA  incident  report  form  was  developed  specifically  for  the  research  study,  to  be  completed 
whenever  an  instructor  observed  a  decision-making  problem  of  significance  to  mission  performance 
or  safety,  and  was  designed  to  be  short  enough  for  on-the-spot  completion.  The  incident  report 
(Appendix  A)  provided  space  for  the  observer  to  enter  a  brief  narrative  description  of  the  event,  as 
well  as  to  specify  characterizing  data  such  as  leadership  level  or  mission,  and  contributing  factors 
such  as  fatigue,  training,  or  equipment  condition.  Experience  with  the  incident  report  form  led  to  a 
parallel  effort  to  distribute  the  form  to  units  deployed  to  Iraq.  Introduction  letters  and  completion 
instructions  were  included  with  each  form,  which  could  be  returned  voluntarily  and  anonymously  by 
prepaid  mail. 

The  Marine  training  staffs  also  utilized  an  evaluation  form  of  their  own.  These  forms  were 
completed  as  part  of  their  instructor  duties  and  addressed  critical  factors  of  planning,  preparation,  and 
execution  for  each  mission,  with  space  for  additional  instructor  comments  as  needed.  To  supplement 
the  final  data  set,  these  forms  were  reviewed  and  scored  with  the  same  protocol  used  for  evaluation 
of  the  incident  report  forms. 

PROCEDURE 

Command  leadership  was  first  briefed  regarding  the  purpose  of  the  experiment  and  the  desired 
data  collection  procedures.  After  official  endorsement  was  obtained  for  the  study,  the  staff  was 
briefed  on  the  specific  data  collection  method.  The  incident  report  form  was  explained  in  detail, 
including  the  importance  of  anonymity  for  the  people  under  observation.  Instructors  with  operational 
experience  were  also  encouraged  to  complete  incident  reports,  as  time  permitted,  pertaining  to 
recollections  of  SA  performance  errors  or  other  decision-making  problems  observed  during  their  own 
deployments.  After  all  questions  were  answered,  a  supply  of  incident  report  forms  was  left  with  the 
command,  and  the  training  staff  went  on  to  conduct  their  normal  duties. 

At  the  conclusion  of  each  unit  training  cycle  (approximately  every  2  weeks),  the  experimenters 
returned  to  collect  the  incident  report  forms,  including  duplicates  of  available  instructor  training 
evaluation  forms.  Reports  that  contained  irrelevant  or  uninterpretable  events  were  first  discarded. 
Remaining  forms  were  then  checked  to  remove  any  personally  identifiable  information,  and  collated 
to  remove  any  redundant  reports  (i.e.,  multiple  sources  reporting  on  the  same  incident).  Forms 
returned  from  Iraq  were  scored  as  received,  until  the  end  of  the  data  collection  period.  Finally,  all 
data  were  entered  into  an  Excel  spreadsheet  for  analysis. 

Each  incident  was  then  classified  according  to  the  consensus  judgment  of  the  research  team  as 
primarily  representing  SA  Level  1,  2,  or  3  characteristics,  using  the  Endsley  model.  Although  many 
reports  contained  elements  that  could  legitimately  reflect  more  than  one  SA  category,  only  a  single 
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classification  was  assigned  to  each  incident  for  this  initial  analysis.  In  addition,  data  from  special 
response  fields  were  extracted  from  each  report,  as  follows: 

•  Source  -  Training  or  operational  Deployment 

•  Mission,  using  the  SASO  model,  above 

•  Leadership  Level  -  the  individual  most  responsible  (or  involved)  in  the  incident,  including  Fire 
Team  Leader,  Squad  Leader,  Platoon  Leader,  and  Company  Commander 

•  Contributing  Factors  -  conditions  or  items  that  were  involved  in  the  incident,  which  were 
further  parsed  into 

o  Standing  Conditions,  involving  characteristics  that  were  fixed  or  that  did  not  change 
significantly  during  operations,  such  as  experience,  pre-existing  training  or  skill  levels, 
equipment,  and  communications  support 

o  Acute  Conditions,  involving  characteristics  that  could  fluctuate  during  operations,  such  as 
daily  fatigue  or  sleep,  stress  or  overload,  and  level  of  inattention  or  complacency 

PERSONNEL 

Reports  were  gathered  from  units  completing  SASO  training,  Mojave  Viper  training,  or  on 
deployment  in  Iraq  between  May  2005  and  September  2008. 

ANALYSIS 

This  study  was  conducted  to  explore  the  potential  of  applying  a  theoretical  model  of  situation 
awareness  to  a  collection  of  operational  events.  Data  were  presented  in  terms  of  bar  charts  and 
frequency  tables  to  reveal  SA  patterns  across  the  two  primary  factors  of  interest — Leadership  Level 
and  Mission.  Analyses  for  this  initial  study  were  primarily  descriptive. 

Inferential  statistical  analysis  was  not  applied  to  the  data  set  because  of  the  disparity  in  numbers  of 
reports  from  Training  and  Deployed  sources  and  the  small  sample  sizes  that  resulted  from  even  basic 
partitioning  of  the  data  set.  Analytical  approaches  for  future  data  collection  efforts  are  presented  in 
the  Discussion. 
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RESULTS 


A  total  of  79  reports  were  obtained  from  the  training  environment  and  22  reports  were  obtained 
from  operational  deployments  to  Iraq  at  the  conclusion  of  the  data  collection  period,  yielding  a  total 
of  101  distinct  incidents. 

GENERAL 

Figure  1  shows  the  breakdown  of  reports  according  to  the  Endsley  model,  sorted  by  Source.  All 
three  data  sets — Training,  Deployed,  and  Combined — show  the  same  step-wise  structure,  i.e.,  Level 
1  errors  >  Level  2  errors  >  Level  3  errors.  Based  on  these  results,  the  majority  of  incidents  were 
attributable  to  a  failure  to  perceive  important  features  of  the  environment  or  situation. 


Data  Source 


Figure  1 .  Data  Source  x  SA  Level. 


Table  1  shows  the  breakdown  of  reports  by  Mission.  Most  reports  came  from  Urban  Assault, 
Patrol,  and  Checkpoint  Operations.  It  is  possible,  but  not  conclusive,  that  relative  numbers  of 
incident  reports  reflect  characteristics  of  the  missions  themselves  (such  as  exposure  to  hazards  for  a 
protracted  period,  activities  performed  while  dismounted,  etc.)  and  therefore  provide  an  ordinal 
predictor  of  mission  complexity. 

Table  1.  Data  Source  x  Mission. 


Source 

Training 

Deployed 

Combined 

Patrol 

22  5 

27 

Mounted  Patrol 

82 

10 

Cordon  /  Knock 

13  0 

13 

Urban  Assault 

19  9 

28 

Checkpoint 

144 

18 

Convoy 

0  1 

1 

Unknown 

0  1 

1 

FEX 

20 

2 

Firm  Base 

1  0 

1 
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The  same  three  missions — Urban  Assault,  Patrol,  and  Checkpoint  Operations — dominated  the  data 
set  when  expressed  as  proportions  (Figure  2),  with  Urban  Assault  (planned  and  executed  as  an 
offensive  operation)  accounting  for  the  most  Deployed  incidents. 
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Data  Source 

Figure  2.  Data  Source  x  Mission. 

In  the  results  that  follow,  combined  sources  are  presented  first,  followed  by  results  from  Training 
and  results  from  Deployment  settings.  The  analysis  of  Contributing  Factors  is  presented  last.  In 
addition,  each  analysis  is  shown  from  two  complementary  perspectives  to  reflect  the  potential 
understanding  to  be  gained  from  this  approach  to  performance  modeling: 

•  The  pattern  of  SA  error  frequencies  across  a  factor  (e.g.,  the  relative  proportion  of  SA  Level  1, 
2,  and  3  errors  for  Leadership  Level),  and 

•  The  pattern  of  SA  error  frequencies  within  each  level  of  the  factor  (e.g.,  the  relative  proportion 
of  SA  Level  1,  2,  and  3errors  for  the  Fire  Team  Leader,  for  the  Squad  Leader,  for  the  Platoon 
Leader,  and  for  the  Company  Commander). 

Because  small  sample  sizes  in  many  categories  yielded  dramatic  differences  in  raw  data  counts, 
subsequent  analyses  are  presented  in  tenns  of  relative  proportions  within  each  category  (i.e.,  the 
approach  of  Figure  2),  which  appeared  to  provide  more  stable  and  interpretable  results. 

PART  I  -  LEADERSHIP  LEVEL 

Distributed  operations  (DO)  doctrine  places  more  reliability  on  junior  leaders  to  execute  decision¬ 
making  tasks  formerly  conducted  by  more  senior  leaders.  It  was  useful,  therefore,  to  investigate  SA 
incidents  at  different  levels  of  leadership  in  order  to  characterize  current  performance  levels  and  the 
direction  that  future  DO  training  may  need  to  take. 

Combined  Results 

As  shown  in  Figure  3a,  the  highest  proportion  of  errors  occurred  at  SA  Level  1  (Perception),  in 
keeping  with  earlier  data  depictions.  The  data  transposition  of  Figure  4  shows  that  the  highest  error 
rates  were  observed  among  Squad  and  Platoon  Leaders  at  all  three  SA  levels.  This  may  reflect  the 
relative  responsibilities  of  these  leadership  levels  or  may  indicate  that  Fire  Team  Leaders  do  not  have 
the  decision-making  duties  that  would  lead  to  the  type  of  incidents  that  triggered  the  reports  of  this 
data  set.  The  result  for  Company  Commander  is  more  difficult  to  interpret;  it  is  possible  that  this 
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leadership  role  has  sufficient  time  for  decision  making,  or  access  to  information  support  resources, 
that  resulted  in  relatively  few  reported  errors. 

Figure  3b  shows  an  interesting  pattern  of  SA  errors  for  the  Squad  Leader,  with  most  errors 
occurring  at  Level  3  (Projection).  This  is  contrasted  with  the  pattern  of  Figure  1,  which  showed  a 
generally  decreasing  frequency  of  errors  with  higher  SA  levels,  and  may  be  indicative  of  the 
challenges  faced  by  this  junior,  yet  pivotal  leadership  position. 


Combined  Reports 


SA  Level 


SA  Level  1 

SA  Level  2 

SA  Level  3 
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Fire  Team 

72 

1 

10 

Squad 

24 

20 
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22  13 
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Company 

4  1 

1 

6 

(a)  Leadership  Level  x  SA  Errors  -  Combined 


Combined  Reports 
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SA  Level  3 
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8 

(b)  Leadership  Level  x  SA  Errors  -  Combined 

Figure  3.  SA  Errors:  Different  Levels  of  Leadership  -  Combined. 
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Training  Results 

The  pattern  of  SA  errors  observed  from  Training  (Figure  4a),  and  their  breakdown  by  Leadership 
Level,  essentially  matches  those  of  the  Combined  reports  presented  in  Figure  1 — the  vast  majority  of 
reported  errors  occurred  at  SA  Level  1.  The  predominance  of  perception  errors  (i.e.,  detecting 
elements  of  the  environment)  is  possibly  due  to  the  fact  that  Training  reports  made  up  the  majority  of 
the  data  set  and  Marines  are  still  mastering  tactical  skills  at  this  stage  of  their  development. 

Figure  4b  also  shows  the  same  reversed  pattern  for  squad  leader  performance  found  earlier,  again 
probably  due  to  the  major  contribution  of  Training  reports  to  the  data  set. 
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(a)  Leadership  Level  x  SA  Errors  -  Training 


Training  Reports 
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(b)  SA  Errors  X  Leadership  level  -  Training 

Figure  4.  SA  Errors:  Different  Levels  of  Leadership  -  Training. 
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Deployment  Results 

Differences  in  error  patterns  between  Deployment  and  Training  data  (Figures  4  and  5)  may  be  due 
to  the  disparity  in  numbers  of  reports  received  from  these  sources.  Deployment  data  were  relatively 
sparse  and,  therefore,  frequency  counts  were  probably  less  stable.  Figure  5b,  however,  also  shows  the 
reversed  pattern  of  SA  errors  for  the  Squad  Leader  role. 


Deployed  Reports 
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(a)  Leadership  Level  x  SA  Errors  -  Deployed 


Deployed  Reports 
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(b)  SA  Errors  x  Leadership  level  -  Deployed 

Figure  5.  SA  Errors:  Different  Levels  of  Leadership  -  Deployed. 
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In  summary,  results  for  analyses  of  Leadership  Level  indicate  that: 

•  SA  error  frequencies  follow  an  overall  pattern,  with  more  Level  1  than  Level  2  errors,  and 
more  Level  2  than  Level  3  errors.  The  predominance  of  this  effect  in  the  Training  data  may 
indicate  that  experience  may  account  for  much  of  this  SA  error  category. 

•  Most  incidents  involved  the  Squad  Leader,  followed  by  the  Platoon  Leader,  the  Tire  Team 
leader  and  Company  Commander,  respectively. 

•  The  pattern  of  SA  errors  was  reversed  for  the  Squad  Leader,  compared  with  the  data  set  as  a 
whole,  with  more  Level  3  (Projection)  errors  than  Level  1  (Perception)  errors,  and  may 
indicate  special  conditions  of  this  position. 

PART  II  -  MISSION 

Each  SASO  mission  reflects  different  degrees  of  complexity  and  hazard,  depending  on  the  amount 
of  time  spent  in  a  hostile  area,  the  number  of  people  involved  (i.e.,  military  and  civilian,  friendly,  and 
hostile),  the  availability  of  protective  cover,  and  the  clarity  of  task  objectives.  Data  evaluation  must 
therefore  include  such  contextual  factors. 

Note  that  one  report,  obtained  from  a  SASO  instructor  regarding  a  deployment  experience, 
contained  valuable  and  relevant  data  regarding  SA  performance.  Unfortunately,  the  specific  mission 
was  not  listed  on  the  report.  Rather  than  delete  the  incident,  an  “Unknown”  category  was  included  in 
the  results,  below,  although  little  could  be  gleaned  from  the  report. 

Combined  Results 

As  noted  earlier,  the  Urban  Assault,  Patrol,  and  Checkpoint  missions  accounted  for  most  of  the 
incident  reports,  and  that  these  missions  broke  down  across  the  three  SA  levels  in  a  stepwise  fashion. 
Figure  6a  completes  the  picture  of  relative  error  frequencies  for  all  mission  types  and  all  SA  levels. 

While  the  stepwise  relationship  in  error  frequency  across  Level  1,  Level  2,  and  Level  3  is  roughly 
maintained  in  this  breakdown,  it  is  apparent  in  both  Figure  6a  and  6b  that  some  missions  showed 
widely  differing  effects  in  this  regard,  likely  due  to  the  instability  of  very  low  frequency  counts  (e.g., 
Firm  Base),  which  influenced  the  proportions  shown  in  the  bar  graphs.  Relationships  across  SA 
levels  are  easier  to  see  in  the  accompanying  table  of  each  figure. 

Note  that  Convoy,  Firm  Base,  and  FEX  differed  markedly  from  other  missions  in  the  number  of 
incident  reports  collected.  Certainly,  the  FEX  mission  is  unique  to  Training,  and  low  report  numbers 
are  explainable  on  that  basis,  while  the  Firm  Base  mission  may  be  approached  differently  during 
deployment.  Further  analysis  and/or  data  collection  is  necessary,  however,  to  resolve  whether  the 
Convoy  disparity  is  due  to  the  characteristics  of  the  mission  (e.g.,  greater  team  cooperation,  more 
structured  event  sequences,  etc.),  to  its  relative  frequency  of  occurrence  in  training  syllabi  or 
deployed  assignments,  or  to  some  other  factor. 

Training  Results 

Although  the  relative  pattern  of  SA  errors  (Level  1  >  Level  2  >  Level  3)  can  still  be  seen  across 
missions  where  reports  were  provided  (Figure  7a),  these  patterns  vary  dramatically  by  Mission 
(Figure  7b).  No  reports,  or  extremely  small  numbers  of  reports,  were  received  for  several  missions 
(as  discussed  for  Combined  results,  above),  making  interpretation  problematic.  Again,  the  breakdown 
of  SA  errors  recorded  from  training  events  is  easier  to  see  in  the  frequency  tables  of  Figure  7  than 
from  the  bar  graphs. 
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Combined  Reports 
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(b)  SA  Errors  x  Mission  Type  -  Combined 

Figure  6.  SA  Errors:  Mission  Type  -  Combined. 
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Training  Reports 
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(a)  SA  Errors  x  Mission  Type  -  Training 

Figure  7.  SA  Errors:  Mission  Type  -  Training. 
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Deployment  Results 

The  low  numbers  of  reports  from  deployment,  shown  In  Figure  8,  give  caution  to  any 
interpretation.  While  the  characteristic  relationship  across  SA  levels  was  again  found  for  the  mission 
with  the  highest  report  count  (Urban  Assault),  other  mission  data  were  too  sparse  to  support  further 
conclusions. 
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Figure  8.  SA  Errors:  Mission  Type  -  Deployed. 
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In  summary,  examination  of  SA  error  distributions  by  Mission  indicated  that: 

•  The  Urban  Assault,  Patrol,  and  Checkpoint  Operations  missions  accounted  for  most  of  the  SA 
errors.  This  may  be  due  to  the  inherent  characteristics  of  these  mission,  e.g.,  complexity, 
duration,  or  hazard,  but  the  causal  factors  could  not  be  resolved  solely  from  these  data. 

•  The  parsing  of  such  a  limited  data  set  into  such  a  large  number  of  missions  meant  that  most 
Mission  categories  were  represented  by  extremely  low  numbers  of  reports,  or  none  at  all. 
Clearly,  a  limit  is  reached  for  the  number  of  groups  that  can  be  analyzed  with  the  size  of  the 
current  data  set. 

PART  III  -  CONTRIBUTING  FACTORS 

Because  operational  performance  is  a  function  of  multiple  factors,  the  incident  report  forms  used 
for  this  study  included  response  items  for  denoting  some  typical  influences  on  military  decision 
making.  The  objective  was  to  amplify  the  narrative  of  an  incident  and  to  place  it  in  a  larger  military 
context;  characterizing  the  impact  of  contributing  factors  to  SA  can  provide  an  enhanced  foundation 
for  directing  improvement  interventions.  Two  classes  of  conditions  were  addressed:  Standing  and 
Acute. 

Standing  conditions  were  defined  as  major  elements  of  the  Marine  environment  that  were  brought 
to  a  mission  such  as  Equipment,  Communications,  and  Training.  Note  that  formal  training  was 
equated  with  skill  level  for  this  study,  and  were  distinguished  from  Experience,  which  could  be 
accumulated  independently  from  Training.  This  distinction  is  important  to  evaluation  of  the  results, 
and  may  require  reconsideration  in  future  data  collection  efforts. 

Acute  conditions  fluctuate  at  shorter  time  scales  and  include  such  factors  as  Fatigue,  lack  of  sleep, 
Stress,  overload,  Inattention,  and  complacency.  Acute  conditions  are  often  under  more  individual  or 
unit  leader  control.  The  design  of  the  incident  report  form  involved  factor  combinations,  i.e.,  Fatigue 
/  Sleep,  Stress  /  Overload,  and  Inattention  /  Complacency,  which  may  have  sacrificed  definitional 
precision  for  speed  and  convenience  in  completing  the  form,  and  which  may  also  require 
reconsideration  in  future  studies. 

Note  that  factor  counts  can  exceed  total  incident  report  counts,  as  more  than  one  factor  could  be 
listed  for  each  incident. 

Standing  Conditions 
Combined  Results 

As  shown  in  Figure  9,  all  factors  except  Equipment  showed  stable  (and  roughly  equivalent) 
patterns  of  SA  errors.  Equipment  reports  represented  only  2%  of  all  data,  but  most  reports  were 
collected  from  the  training  environment  where  equipment  resources  were  either  not  taxed  or 
equipment  was  not  available  (and  could  not,  therefore,  be  a  factor  in  SA  incidents). 

Virtually  all  Communications  reports  involved  verbal  interactions,  i.e.,  close-range  or  face-to-face 
commands.  Nevertheless,  the  role  of  such  communications  in  supporting  or  impeding  good  SA 
illuminates  similar  issues  with  radio  and  other  IT  communications  systems.  Note  the  heavy 
involvement  of  Communications  as  a  factor  in  SA  Level  1  (Perception)  errors.  The  consistent  trend 
regarding  the  Level  1  -  Level  2  -  Level  3  relationship  is  maintained  throughout  this  analysis,  as  well. 

Experience  showed  the  highest  proportion  of  Level  3  (Projection)  errors  for  all  of  the  conditions, 
which  may  indicate  a  significant  role  for  experience  in  making  effective  decisions  or  in  establishing 
deeper  levels  of  situation  awareness. 
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Figure  9.  SA  Errors:  Standing  Conditions  -  Combined. 
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Training  Results 

Training  data  results  (Figure  10)  showed  few  differences  from  the  Combined  data  results.  Again, 
no  equipment-related  incidents  were  found  during  Training,  which  is  most  likely  a  function  of  the 
controlled  nature  of  the  syllabus  (i.e.,  where  resources  were  either  prepared  in  advance  of  the  training 
event  or  were  not  used  at  all). 
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Figure  10.  SA  Errors:  Standing  Conditions  -  Training. 
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Note  that  Communications  contributed  most  strongly  to  SA  Level  1  (Perception)  errors,  indicating 
the  critical  nature  of  basic  information  exchange  to  an  awareness  of  the  environment.  In  addition, 
Experience  played  a  very  small  role  in  SA  Level  3  (Projection)  errors,  which  could  indicate  a  ceiling 
effect  of  experience  on  performance  under  Training  conditions  (i.e.,  which  may  not  require 
significant  application  of  SA  Level  3  (Projection)  performance),  or  a  relative  paucity  of  experience 
among  Marines  at  this  stage  of  development  (i.e.,  in  which  case  it  would  not  represent  a  significant 
basis  for  performance  assessment). 

Deployment  Results 

The  Deployment  data  shown  in  Figure  1 1  are  consistent  with  the  patterns  found  for  Training  data. 
Note  the  consistent  role  of  Experience  at  each  level  of  SA  for  this  data  source,  with  increased 
involvement  of  Level  3  SA.  While  it  is  tempting  to  attribute  the  importance  of  Experience  to  the 
demands  of  deployed  operations,  the  sample  numbers  are  too  low  to  draw  such  firm  conclusions. 

Figure  1 1  also  shows  the  relative  greater  contribution  of  Standing  Conditions  to  SA  Level  1 
(Perception)  errors  in  the  Deployed  environment,  when  compared  with  Training  results. 

A  review  of  Standing  factors  associated  with  the  data  set  indicated  that: 

•  Experience  was  a  prominent  factor  in  SA  errors,  especially  in  the  reports  gathered  from  the 
Deployed  environment,  with  SA  Level  3  errors  accounting  for  a  larger  proportion  of  incidents 
compared  to  the  Training  environment.  Although  this  result  may  show  an  interesting  feature  of 
an  SA  modeling  approach  to  empirical  data,  respondents  may  have  also  interpreted 
“experience”  in  a  broad  way.  This  observation  therefore  remains  tentative,  pending  further  data 
collection. 

•  Standing  Conditions  contributed  primarily  to  SA  Level  1  errors;  this  proportion  was  greater  in 
the  Deployed  setting  than  in  the  Training  setting. 

•  Equipment  factors  played  almost  no  role  in  SA  performance  in  the  Training  setting,  possibly 
due  to  controlled  conditions  unique  to  that  environment. 
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(a)  Standing  Condition  x  SA  Errors  -  Deployed 
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(b)  SA  Errors  x  Standing  Condition  -  Deployed 


Figure  1 1.  SA  Errors:  Standing  Conditions  -  Deployed. 
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Acute  Conditions 


Combined  Results 

The  tables  in  Figure  12  illustrate  that  the  major  factors  involved  in  SA  incidents  were  Stress  / 
Overload  and  Inattention  /  Complacency.  The  relatively  short  2-week  duration  of  the  training 
syllabus  for  SASO  or  Mojave  Viper  may  account  for  the  relatively  mild  impact  of  Fatigue  /  Sleep.  Of 
these  factors,  Inattention  /  Complacency  was  associated  most  prominently  with  SA  Level  1 ,  while 
Stress  /  Overload  was  most  associated  with  SA  Level  2. 

Characteristic  patterns  of  SA  ratios  (i.e.,  Level  1  >  Level  2  >  Level  3)  can  again  be  seen  in  Figure 
12  for  Acute  Conditions,  i.e.,  with  primary  influence  of  these  factors  found  for  SA  Level  1 . 

Training  Results 

The  minor  role  of  Fatigue  /  Sleep  on  incidents  from  the  Training  environment  can  again  be  seen  in 
Figure  13.  The  effects  of  Stress  /  Overload  on  both  Level  1  and  Level  2  SA  incidents  can  also  be  seen 
in  the  figure,  which  indicates  that  this  factor  influenced  both  Perception  and  Comprehension 
performance  in  the  incident  reports. 

It  is  noteworthy  that  62%  (21  of  34)  of  Level  1  errors  were  attributed  to  Inattention  / 

Complacency,  compared  with  only  33%  (1 1  of  34)  attributed  to  Stress  /  Overload.  This  condition  is 
reversed  for  Level  2  errors,  however — 5  of  17  (29%)  and  11  of  17  (65%),  respectively — implying  a 
stronger  contribution  of  Inattention  /  Complacency  to  Level  1  errors  (the  perception  of  elements  in 
the  environment)  and  a  stronger  contribution  of  Stress  /  Overload  to  Level  2  errors  (the 
comprehension  of  their  meaning). 
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Combined  Reports 
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(b)  SA  Errors  x  Acute  Condition  Combined 


Figure  12.  SA  Errors:  Acute  Conditions  -  Combined. 
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Training  Reports 
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(b)  SA  Errors  x  Acute  Condition  -  Training 

Figure  13.  SA  Errors:  Acute  Conditions  -  Training. 
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Deployment  Results 

SA  patterns  were  more  difficult  to  detect  in  the  Deployed  data,  shown  in  Figure  14,  although 
Inattention  /  Complacency  was  associated  with  more  SA  incidents  than  Fatigue  /  Sleep  or  Stress  / 
Overload. 
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(b)  SA  Errors  x  Acute  Condition  -  Deployed 

Figure  14.  SA  Errors:  Acute  Conditions  -  Deployed. 
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Although  the  number  of  reports  was  low  for  this  evaluation,  the  figure  shows  the  primary  locus 
(Level  1)  for  associations  with  these  Acute  Conditions  in  reports  gathered  from  deployed  forces. 
Overall,  evaluation  of  Acute  Conditions  revealed  that: 

•  Inattention  /  Complacency  was  most  associated  with  Level  1  (Perception)  SA  errors,  and 
played  the  largest  role  in  both  Training  and  Deployed  settings 

•  Stress  /  Overload  was  associated  with  both  SA  Level  1  and  Level  2  errors,  at  least  in  the 
Training  setting 

•  Fatigue  /  Sleep  did  not  play  a  significant  role  in  performance  from  the  Training  environment, 
probably  because  of  the  brief  (e.g.,  2-week)  duration  of  the  syllabus 
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DISCUSSION 


While  the  work  reported  here  was  only  exploratory,  this  first-generation  incident  report  appeared 
to  be  effective  as  a  data  collection  tool  that  could  be  used  with  little  training  by  both  expert 
evaluators  and  operational  Marines,  as  part  of  their  normal  duties.  While  incident  report  data  from 
operational  settings  are  essential  for  comparison  with  training  data,  a  more  systematic  approach  to 
the  collection  of  such  field  data  is  required  in  the  future.  Certainly,  greater  equivalence  in  the 
numbers  of  reports  collected  from  operational  and  training  environments  is  essential  to  reliable 
analysis.  Furthermore,  although  retrospective  reports  from  operational  deployments  proved  useful  to 
this  study,  the  lack  of  control  over  what  was  provided,  the  varying  amounts  of  time  since  the  events 
occurred,  and  the  motivations  of  the  people  who  furnished  these  reports  left  open  the  issue  of  survey 
rigor.  Finally,  the  incident  report  form  itself  could  benefit  from  a  more  considered  selection  of 
contributing  factors  to  ensure  that  more  precisely  defined  information  is  gathered. 

The  SA  scoring  assignment  procedure  used  for  this  study  (i.e.,  Level  1,  Level  2,  or  Level  3)  was  a 
single  classification  decision  made  for  each  incident  report.  A  method  for  accommodating  the  inputs 
of  multiple  evaluators,  however,  (i.e.,  beyond  the  consensus  approach  employed  here)  would  provide 
the  data  variance  that  could  support  inferential  statistical  analysis.  Experience  with  the  scoring 
process  also  showed  that  many  reports  contained  descriptions  of  multiple  activities  that  contributed 
to  the  final  event  outcome.  A  procedure  to  integrate  these  various  activities  into  a  common 
theoretical  rubric  was  not  developed  in  this  study.  It  is  likely  that  a  more  comprehensive  scoring 
protocol  that  includes  the  sequence  and  timing  of  these  activities  will  be  necessary,  however,  to 
support  rigorous  SA  evaluations  of  real  world  incidents  such  as  those  used  in  this  study. 

The  experience  gained  from  this  study  has  also  shown  that  both  the  incident  report  content  and  the 
theoretical  perspective  could  benefit  from  further  development.  Consideration  of  more  precise  factor 
definitions  (e.g.,  between  Fatigue  and  Sleep),  for  example,  and  comparative  evaluations  of  alternative 
SA  models  such  as  the  OODA  loop  (e.g.,  Brehmer,  2004),  could  further  enhance  both  the  methods 
and  the  knowledge  needed  to  enhance  Marine  Corps  training.  Both  of  these  efforts  could  be  readily 
achieved  by  reviews  of  existing  data  bases  (e.g.,  those  maintained  by  the  Marine  Corps  Center  for 
Lessons  Learned),  to  generate  the  kinds  of  factor  analysis  dimensions  that  could  refine  the  items  of 
interest  during  future  field  research. 

A  theoretical  weakness  of  the  critical  incident  approach  is  time.  The  data  reported  here  were 
collected  over  a  period  of  years.  Operational  demands,  political  conditions,  and  military  doctrine, 
however,  can  change  relatively  rapidly.  An  analysis  approach  based  only  on  collection  of  empirical 
field  data  contains  the  risk  of  being  rendered  irrelevant  by  the  time  results  are  processed  and 
available  for  use.  By  comparison,  the  Aviation  Safety  Reporting  System  (ASRS),  which  was  earlier 
used  as  a  conceptual  guide  for  the  critical  incident  method,  is  concerned  with  the  relatively  slow- 
changing  domain  of  civil  aviation  operations.  A  solution  to  this  situation  is  to  maintain  a  data 
collection  program  on  a  continuing  basis,  which  would  then  better  track  such  operational  shifts  by 
reducing  response  latencies,  as  the  most  current  data  would  be  continually  available  for  analysis. 
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CONCLUSIONS 


The  primary  goal  of  this  study  was  determine  whether  useful,  coherent,  and  model-based 
information  could  be  extracted  from  observations  of  real-world  events.  More  specifically,  the 
objective  was  to  characterize  situation  awareness  (SA)  errors  of  small  unit  leaders  under  realistic 
conditions,  where  such  errors  led  to  observably  bad  outcomes,  and  to  determine  the  relationships  of 
leadership  level,  tactical  mission,  and  other  contributing  factors  to  those  errors.  We  sought  to 
structure  SA  performance  reports  according  to  the  Endsley  model,  using  field  reports  from  both 
training  and  deployed  settings.  While  the  number  of  incident  reports  was  small,  the  results 
demonstrated  that  it  is  possible  to  unobtrusively  capture  systematic  performance  data  that  are 
meaningful  to  both  Marines  and  researchers.  Based  on  these  results,  it  appears  that  such  data — in 
larger  quantities — can  support  the  generation  of  interpretable,  quantitative  patterns  of  SA 
performance  that  could  be  used  to  guide  interventions  in  training,  technology  support,  and  tactics. 
More  importantly,  the  use  of  event  reporting  as  a  research  tool  ensures  that  modeling  and  design 
efforts  can  be  placed  on  a  foundation  of  empirically  documented  problem  factors,  rather  than  only  on 
an  extrapolation  of  structured  research  studies. 

Based  on  the  data  collected  to  this  point,  it  appears  that: 

1 .  Interpretable  (i.e.,  theory-related)  dimensions  of  SA  can  be  characterized  through  the  incident 
report  method,  but  large  amounts  of  data  are  required  for  stability  of  the  results. 

2.  The  Endsley  model  of  situation  awareness  appeared  to  provide  useful  dimensions  for 
evaluating  performance  across  a  wide  variety  of  missions  and  leadership  levels  (although  other 
models  should  be  explored,  as  well). 

3.  Although  small,  differences  between  Training  and  Deployed  settings  were  found,  further  data 
collection  may  yet  uncover  important  differences  in  SA  patterns  between  these  two 
environments  that  may  provide  important  guidance  for  training  improvement.  Data  collection 
from  both  settings,  therefore,  should  continue. 
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APPENDIX  A:  INCIDENT  REPORT 


CSA  Incident  Report 


Who?  (Instructors  asked  to  identify  only  the  leadership  level  of  the  person  responsible  for  the 
incident,  i. e. ,  no  personally-identifiable  information) 


What  happened?  (Instructors  asked  to  provide  brief  narration  of  event  and  its  outcome ) 


When  (in  the  timeline)?  (Instructors  asked  to  state  whether  incident  occurred  during 
planning,  execution,  etc.) 


Why  did  it  happen?  (Instructors  asked  to  provide  their  own  opinions  about  the  causes  or 
event-sequence  that  led  to  the  incident) 
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